Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary.

Note: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a write up template that can be used to guide the writing process. Completing the code template and writeup template will cover all of the rubric points for this project.

The rubric contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 0: Load The Data

In [1]:
enable_aug = False

# Load pickled data
import pickle

# TODO: Fill this in based on where you saved the training and testing data
training_file = "./traffic-signs-data/train.p"
validation_file="./traffic-signs-data/valid.p"
testing_file = "./traffic-signs-data/test.p"

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)

X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']

Step 1: Dataset Summary & Exploration

The pickled data is a dictionary with 4 key/value pairs:

  • 'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
  • 'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
  • 'sizes' is a list containing tuples, (width, height) representing the original width and height the image.
  • 'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the pandas shape method might be useful for calculating some of the summary results.

Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas

In [2]:
### Replace each question mark with the appropriate value. 
### Use python, pandas or numpy methods rather than hard coding the results
import numpy as np

# TODO: Number of training examples
n_train = X_train.shape[0]

# TODO: Number of testing examples.
n_valid = X_valid.shape[0]

# TODO: Number of testing examples.
n_test = X_test.shape[0]

# TODO: What's the shape of an traffic sign image?
image_shape = X_train.shape[1:3]

# TODO: How many unique classes/labels there are in the dataset.
n_classes = np.unique(y_train)

print("Number of training examples =", n_train)
print("Number of valid examples =", n_valid)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 34799
Number of valid examples = 4410
Number of testing examples = 12630
Image data shape = (32, 32)
Number of classes = [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42]

Include an exploratory visualization of the dataset

Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.

The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.

NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.

In [3]:
### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline

## import name of each class
import csv
classes_string = []
with open('signnames.csv', newline='') as csvfile:
    rd = csv.reader(csvfile, delimiter=',', quotechar='|')    
    i = 0
    for row in rd:
        if i==0:
            i += 1
            continue
        else:
            classes_string.append(row[1])

fig, ax = plt.subplots(figsize=(16,4))
n_bins = 43
ax.hist(y_train,n_bins, histtype='bar', color='green', label='train')
ax.hist(y_test,n_bins, histtype='bar', color='blue', label='test')
ax.hist(y_valid,n_bins, histtype='bar', color='magenta', label='valid')
ax.set_xticks(range(n_bins))
ax.set_xticklabels(classes_string, rotation=40,ha='right')
ax.set_title('Data ditribution (train, test, valid)')
plt.legend(['train','test','valid'])
Out[3]:
<matplotlib.legend.Legend at 0x7bf2940>

Sample visualization of each class

In [7]:
# show data distributions of train, valid, test samples
import numpy as np
import matplotlib.pyplot as plt

def GetSampleData(data, labels, n_class):
    import random as rm    
    n_data = data[ labels == n_class ]
    if n_data.shape[0]==0:
        return np.array([])
    else:
        sample = n_data[ rm.randint(0, n_data.shape[0]-1) ]
        return sample.squeeze()

def GetSampleGallary(data, labels, n_class, num):
    if num < 1:
        print('please insert above 1 for num')
        return np.array([])
    gallary = np.array([])
    for i in range(num):
        sample = GetSampleData(data, labels, n_class)        
        if sample.shape[0] != 0:
            if i==0:
                gallary = sample 
            else:
                gallary = np.hstack((gallary,sample))
    return gallary

fig = plt.figure(figsize=(15,80))
for i in np.unique(y_train):
    samples = GetSampleGallary(X_train, y_train, i, 10)
    ax = fig.add_subplot(43, 1, i+1)
    ax.axis('off')
    ax.imshow(samples)
    ax.set_title(classes_string[i],fontsize=12)
plt.show()

Pre-process the Data Set

Five steps for data augmentation and pre-process to improve classification performance

1. Flipping

  • Horizontal flipping: class number 11, 12, 13, 15, 17, 18, 22, 26, 30, 35
  • Vertical flipping: class number 1, 5, 12, 15, 17
  • Horizontal and vertcal flipping: 32, 40
  • Horizontal flipping, but changed class number
    • from: 19,33,36,38,20,34,37,39
    • to : 20,34,37,39,19,33,36,38

2. Rotation

  • Random rotation within +-15 degree was applied to the flipped samples

3. Perspective transform

  • Perspective transform was conducted with 5 iterations
  • Four corners were randomly selected in certain boundaries

4. Y tranformation

  • RGB image was converted to YUV color space
  • Only Y channel was saved for training samples

5. Equalization

  • Histogram equalization was applied to the Y channel
In [229]:
### Function definition for data augmentation

## normalization
def normalized_gray(x, alpha, beta):
    x_mean = np.ceil((x[:,:,:,0] + x[:,:,:,1] + x[:,:,:,2])/3)   
    x = alpha + (x_mean - 0) / (255 - 0 ) * (beta - alpha)
    x.shape = (x.shape[0], 32, 32, 1)
    return x

def intensity_normalization(x):
    import cv2
    
    dst = []
    for img in x:
        img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        img_gray = img_gray / (np.max(img_gray))
        img_gray.shape = (32,32,1)
        dst.append(img_gray)
        
    dst = np.array(dst)
    return dst

def convertRGBtoY(x):
    x = 0.299 * x[:,:,:,0] + 0.587 * x[:,:,:,1] + 0.114 * x[:,:,:,2]     
    x = (x / 255.).astype(np.float32)
    return x

def normalize(x):    
    return (x/255.0).astype('float32')

def equalize_hist(x):
    from skimage import exposure
    import os
    for i in range(len(x)):
        x[i] = exposure.equalize_adapthist(x[i])
        if (i/len(x)*100) % 10 == 0:
            print('Progress: {:.2f}%'.format(i/len(x)*100))
        
    return x

def flip_h(x):
    return x[:,:,::-1,:]

def flip_v(x):
    return x[:,::-1,:,:]

def flip_hv(x):
    return x[:,::-1,::-1,:]

## translation, scaling, rotation, flipping
def flip_data(x, y, labels, cross_labels=None, method=0):    
    _x = np.empty([0,x.shape[1],x.shape[2],x.shape[3]], dtype=x.dtype)
    _y = np.empty([0], dtype=y.dtype)
    
    for i in range(len(labels)):
        x_label = x[y==labels[i]]
        y_label = y[y==labels[i]]
        if method==0: # flip horizontally
            x_label_flipped = flip_h(x_label)
            y_label_flipped = (np.full((len(x_label_flipped)),labels[i])).astype(y.dtype)
        elif method==1: # flip vertically
            x_label_flipped = flip_v(x_label)
            y_label_flipped = (np.full((len(x_label_flipped)),labels[i])).astype(y.dtype)
        elif method==2: # flip both
            x_label_flipped = flip_hv(x_label)
            y_label_flipped = (np.full((len(x_label_flipped)),labels[i])).astype(y.dtype)
        elif method==3: # flip vertically and cross label
            x_label_flipped = flip_h(x_label)        
            y_label_flipped = (np.full((len(x_label_flipped)),cross_labels[i])).astype(y.dtype)
        
        _x = np.append(_x,x_label_flipped, axis=0)
        _y = np.append(_y,y_label_flipped, axis=0)
    
    return _x, _y   

def flip_data_set(x, y, f_list):
    x_flipped = np.empty([0,x.shape[1],x.shape[2],x.shape[3]],dtype=x.dtype)
    y_flipped = np.empty([0],dtype=x.dtype)
    _x, _y = flip_data(x, y, f_list["h"], None, method=0)
    
    x_flipped=np.append(x_flipped,_x,axis=0)
    y_flipped=np.append(y_flipped,_y,axis=0)
    _x, _y = flip_data(x, y, f_list["v"], None, method=1)
    x_flipped=np.append(x_flipped,_x,axis=0)
    y_flipped=np.append(y_flipped,_y,axis=0)
    _x, _y = flip_data(x, y, f_list["hv"], None, method=2)
    x_flipped=np.append(x_flipped,_x,axis=0)
    y_flipped=np.append(y_flipped,_y,axis=0)
    _x, _y = flip_data(x, y, f_list["cs_h"][0,:], f_list["cs_h"][1,:], method=3)
    x_flipped=np.append(x_flipped,_x,axis=0)
    y_flipped=np.append(y_flipped,_y,axis=0)
    
    return x_flipped, y_flipped

def rotate_data(X, ang_margin=15.):
    from skimage.transform import rotate
    from skimage.transform import warp
    from skimage.transform import ProjectiveTransform
    print(X.shape)
    
    _X = np.empty([0,X.shape[1],X.shape[2],X.shape[3]],dtype=X.dtype)
    for i in range(X.shape[0]):        
        img = rotate(X[i], np.random.uniform(-ang_margin, ang_margin), mode = 'edge')        
        img = (img * 255).astype('uint8')
        img.shape = (1,img.shape[0],img.shape[1],img.shape[2])     
        _X = np.append(_X,img, axis=0)        
    return _X

def pespective_transform(X, offset):
    image_size = X.shape[1]    
    
    _X = np.empty([0,X.shape[1],X.shape[2],X.shape[3]],dtype=X.dtype)
    for i in range(X.shape[0]):
        tl_top = np.random.uniform(-offset, offset)     # Top left corner, top margin
        tl_left = np.random.uniform(-offset, offset)    # Top left corner, left margin
        bl_bottom = np.random.uniform(-offset, offset)  # Bottom left corner, bottom margin
        bl_left = np.random.uniform(-offset, offset)    # Bottom left corner, left margin
        tr_top = np.random.uniform(-offset, offset)     # Top right corner, top margin
        tr_right = np.random.uniform(-offset, offset)   # Top right corner, right margin
        br_bottom = np.random.uniform(-offset, offset)  # Bottom right corner, bottom margin
        br_right = np.random.uniform(-offset, offset)   # Bottom right corner, right margin

        transform = ProjectiveTransform()
        transform.estimate(np.array((
                (tl_left, tl_top),
                (bl_left, image_size - bl_bottom),
                (image_size - br_right, image_size - br_bottom),
                (image_size - tr_right, tr_top)
            )), np.array((
                (0, 0),
                (0, image_size),
                (image_size, image_size),
                (image_size, 0)
            )))
        img = warp(X[i], transform, output_shape=(image_size, image_size), order = 1, mode = 'edge')
        img = (img * 255).astype('uint8')
        img.shape = (1,img.shape[0],img.shape[1],img.shape[2])     
        _X = np.append(_X,img, axis=0)

    return _X
In [171]:
f_list = {"h": np.array([11, 12, 13, 15, 17, 18, 22, 26, 30, 35]),
          "v": np.array([1, 5, 12, 15, 17]),
          "hv": np.array([32, 40]),
          "cs_h": np.array([[19,33,36,38,20,34,37,39],[20,34,37,39,19,33,36,38]])}
valid set: original (4410, 32, 32, 3) / extend (7170, 32, 32, 3)
test set: original (12630, 32, 32, 3) / extend (20460, 32, 32, 3)
train set: original (34799, 32, 32, 3) / extend (56368, 32, 32, 3)

- Data Augementation for Validation Set

In [238]:
X_tt = X_valid
y_tt = y_valid

X_flipped, y_flipped = flip_data_set(X_tt,y_tt,f_list)
X_ext = np.append(X_tt,X_flipped,axis=0)
y_ext = np.append(y_tt,y_flipped,axis=0)
print('[flipped]: original', X_tt.shape, '/ extend', X_ext.shape)

X_rotated = rotate_data(X_ext, 15.)
X_ext = np.append(X_ext,X_rotated,axis=0)
y_ext = np.append(y_ext,y_ext,axis=0)
print('[rotated]: ', X_ext.shape)

_x = np.copy(X_ext)
_y = np.copy(y_ext)
for i in range(5):
    X_aug = pespective_transform(_x, 5)
    X_ext = np.append(X_ext,X_aug,axis=0)
    y_ext = np.append(y_ext,_y,axis=0)
    print('[pespective transform]: ', X_ext.shape)    

X_valid_ext = convertRGBtoY(X_ext)
X_valid_ext = equalize_hist(X_valid_ext)
y_valid_ext = y_ext
print('finished!!')

fig = plt.figure(figsize=(15,80))
for i in np.unique(y_ext):
    samples = GetSampleGallary(X_valid_ext, y_ext, i, 10)
    ax = fig.add_subplot(43, 1, i+1)
    ax.axis('off')
    ax.imshow(samples,cmap='gray')
    ax.set_title(classes_string[i],fontsize=12)
plt.show()

print(X_valid_ext.shape, y_valid_ext.shape)
np.save('X_valid_ext',X_valid_ext)
np.save('y_valid_ext',y_valid_ext)
[flipped]: original (4410, 32, 32, 3) / extend (7170, 32, 32, 3)
(7170, 32, 32, 3)
[rotated]:  (14340, 32, 32, 3)
[pespective transform]:  (28680, 32, 32, 3)
[pespective transform]:  (43020, 32, 32, 3)
[pespective transform]:  (57360, 32, 32, 3)
[pespective transform]:  (71700, 32, 32, 3)
[pespective transform]:  (86040, 32, 32, 3)
C:\Users\SRG\Anaconda3\envs\tensor_flow\lib\site-packages\skimage\util\dtype.py:110: UserWarning: Possible precision loss when converting from float32 to uint16
  "%s to %s" % (dtypeobj_in, dtypeobj))
Progress: 0.00%
Progress: 10.00%
Progress: 20.00%
Progress: 30.00%
Progress: 40.00%
Progress: 50.00%
Progress: 60.00%
Progress: 70.00%
Progress: 80.00%
Progress: 90.00%
finished!!

- Data Augmentation for Training Set and Sample Visualization

In [243]:
X_tt = X_train
y_tt = y_train

X_flipped, y_flipped = flip_data_set(X_tt,y_tt,f_list)
X_ext = np.append(X_tt,X_flipped,axis=0)
y_ext = np.append(y_tt,y_flipped,axis=0)
print('[flipped]: original', X_tt.shape, '/ extend', X_ext.shape)

X_rotated = rotate_data(X_ext, 15.)
X_ext = np.append(X_ext,X_rotated,axis=0)
y_ext = np.append(y_ext,y_ext,axis=0)
print('[rotated]: ', X_ext.shape)

_x = np.copy(X_ext)
_y = np.copy(y_ext)
for i in range(5):
    X_aug = pespective_transform(_x, 5)
    X_ext = np.append(X_ext,X_aug,axis=0)
    y_ext = np.append(y_ext,_y,axis=0)
    print('[pespective transform]: ', X_ext.shape)    

X_train_ext = convertRGBtoY(X_ext)
X_train_ext = equalize_hist(X_train_ext)
y_train_ext = y_ext
print('finished!!')

fig = plt.figure(figsize=(15,80))
for i in np.unique(y_ext):
    samples = GetSampleGallary(X_train_ext, y_train_ext, i, 10)
    ax = fig.add_subplot(43, 1, i+1)
    ax.axis('off')
    ax.imshow(samples,cmap='gray')
    ax.set_title(classes_string[i],fontsize=12)
plt.show()

print(X_train_ext.shape, y_train_ext.shape)
np.save('X_train_ext',X_train_ext)
np.save('y_train_ext',y_train_ext)
[flipped]: original (34799, 32, 32, 3) / extend (56368, 32, 32, 3)
(56368, 32, 32, 3)
[rotated]:  (112736, 32, 32, 3)
[pespective transform]:  (225472, 32, 32, 3)
[pespective transform]:  (338208, 32, 32, 3)
[pespective transform]:  (450944, 32, 32, 3)
[pespective transform]:  (563680, 32, 32, 3)
[pespective transform]:  (676416, 32, 32, 3)
C:\Users\SRG\Anaconda3\envs\tensor_flow\lib\site-packages\skimage\util\dtype.py:110: UserWarning: Possible precision loss when converting from float32 to uint16
  "%s to %s" % (dtypeobj_in, dtypeobj))
Progress: 0.00%
Progress: 50.00%
finished!!
(676416, 32, 32) (676416,)

- Data Augmentation for Test Set and Sample Visualization

In [244]:
X_tt = X_test
y_tt = y_test

X_flipped, y_flipped = flip_data_set(X_tt,y_tt,f_list)
X_ext = np.append(X_tt,X_flipped,axis=0)
y_ext = np.append(y_tt,y_flipped,axis=0)
print('[flipped]: original', X_tt.shape, '/ extend', X_ext.shape)

X_rotated = rotate_data(X_ext, 15.)
X_ext = np.append(X_ext,X_rotated,axis=0)
y_ext = np.append(y_ext,y_ext,axis=0)
print('[rotated]: ', X_ext.shape)

_x = np.copy(X_ext)
_y = np.copy(y_ext)
for i in range(5):
    X_aug = pespective_transform(_x, 5)
    X_ext = np.append(X_ext,X_aug,axis=0)
    y_ext = np.append(y_ext,_y,axis=0)
    print('[pespective transform]: ', X_ext.shape)    

X_test_ext = convertRGBtoY(X_ext)
X_test_ext = equalize_hist(X_test_ext)
y_test_ext = y_ext
print('finished!!')

fig = plt.figure(figsize=(15,80))
for i in np.unique(y_ext):
    samples = GetSampleGallary(X_test_ext, y_test_ext, i, 10)
    ax = fig.add_subplot(43, 1, i+1)
    ax.axis('off')
    ax.imshow(samples,cmap='gray')
    ax.set_title(classes_string[i],fontsize=12)
plt.show()

print(X_test_ext.shape, y_test_ext.shape)
np.save('X_test_ext',X_test_ext)
np.save('y_test_ext',y_test_ext)
[flipped]: original (12630, 32, 32, 3) / extend (20460, 32, 32, 3)
(20460, 32, 32, 3)
[rotated]:  (40920, 32, 32, 3)
[pespective transform]:  (81840, 32, 32, 3)
[pespective transform]:  (122760, 32, 32, 3)
[pespective transform]:  (163680, 32, 32, 3)
[pespective transform]:  (204600, 32, 32, 3)
[pespective transform]:  (245520, 32, 32, 3)
C:\Users\SRG\Anaconda3\envs\tensor_flow\lib\site-packages\skimage\util\dtype.py:110: UserWarning: Possible precision loss when converting from float32 to uint16
  "%s to %s" % (dtypeobj_in, dtypeobj))
Progress: 0.00%
Progress: 10.00%
Progress: 20.00%
Progress: 30.00%
Progress: 40.00%
Progress: 50.00%
Progress: 60.00%
Progress: 70.00%
Progress: 80.00%
Progress: 90.00%
finished!!
(245520, 32, 32) (245520,)
In [253]:
X_test_ext.shape = (len(X_test_ext), 32, 32, 1)
X_valid_ext.shape = (len(X_valid_ext), 32, 32, 1)
X_train_ext.shape = (len(X_train_ext), 32, 32, 1)
np.save('X_test_ext',X_test_ext)
np.save('y_test_ext',y_test_ext)
np.save('X_train_ext',X_train_ext)
np.save('y_train_ext',y_train_ext)
np.save('X_valid_ext',X_valid_ext)
np.save('y_valid_ext',y_valid_ext)